Lecture 03
Lecture 2: Review of data and graphing
- We covered
- How to design a well-organized project
- How to implement good naming conventions
- Controlled vocabulary
- Including units in names
- Create and use metadata effectively
- Build tidy, well-structured spreadsheets
- Understand data repositories
- Create effective visualizations with ggplot2
These are variables - do you know what they mean?
TGW - yep its a thing
ODO - what do you think it is?
NO3 - what is it? Are you sure? Why might you get in legal trouble if you used this?
The objectives:
- Understand why statistics is vital in biology
- Distinguish between different types of biological variables
- Learn about accuracy, precision, and bias in measurements
- Calculate and interpret measures of central tendency (mean, median, geometric mean)
- Calculate and interpret measures of spread (standard deviation, variance, IQR)
- Understand data transformations for skewed distributions
- Visualize descriptive statistics for our data
- Learn how to handle uncertainty in our data
We’ll use a dataset on grayling fish from two different lakes to explore these concepts..
Lecture 3: Populations and Samples
Before we dive into descriptive statistics, let’s clarify some fundamental concepts:
- Population: The entire group of things under consideration; the group for which answers obtained from measurements and statistical analysis are pertinent.
- Sample: A subset of the population that is actually measured.
- Sample unit: The individual thing drawn from the population.
Types of populations: -
- Observational population: Usually finite but may be very large (e.g., head width of all corn earworms in a field) -
- Experimental population: Often conceptually infinite (e.g., all possible goldenrod plants that could receive a specific fertilizer treatment)
Sampling involves
- inference - generalizing from what is observed in the sample to what is present in the population.
- Valid inference requires random sampling.
Lecture 3: Parameters vs. Statistics
It’s important to distinguish between:
- Parameters: True numerical values for a population (usually denoted by Greek letters)
- Statistics: Estimates of parameters based on samples (usually denoted by Roman letters)
For example:
- Population mean (μ) is estimated by sample mean (Y̅)
- Population standard deviation (σ) is estimated by sample standard deviation (s)
The standard deviation formula above includes n-1 in the denominator (rather than n) to provide an unbiased estimate of the population parameter.
Understanding the type of variable you’re working with is essential for selecting appropriate statistics:
Measurement or Quantitative Variables
- Continuous: Any value between extremes of scale is possible (e.g., mass, length)
- Discrete (meristic): Only fixed values (usually integers) between extremes are possible (e.g., bristle number, egg count)
Rank Variables (Ordinal)
- Assign only order, not quantity
- Nothing implied about relative distance between values
Categorical Variables (Qualitative)
- No quantitative information (e.g., male/female, living/dead)
- Some are simplifications of quantitative variables (e.g., color instead of wavelength)
asdfsd
Lecture 3: Accuracy, Precision, and Bias
When taking biological measurements, understanding measurement quality is essential:
- Accuracy: Closeness of measured value to true value
- Precision: Closeness of repeated measurements to each other (repeatability)
- Bias: Systematic departure from the true value
Accuracy is a function of both precision and bias. For statisticians, bias is usually a more serious problem than low precision because:
It’s harder to detect (true value usually unknown)
Low precision can be compensated for by increased sample size
asdfsd
Practice Exercise 1: What are potential sources of error in pine needles or fish?
For our grayling data, potential sources of measurement error might include:
Precision issues:
- Variations in how fish are measured (e.g., slightly bent fish)
Bias issues:
- Systematic underestimation of length if measurements aren’t taken from the true tip of the snout to the end of the tail
Accuracy issues? what could they be?
The two most common measures of central tendency are the mean and the median.
The Arithmetic Mean The arithmetic mean is the average of a set of measurements:
stuff
# Calculate mean length of all fish
mean_length <- mean(grayling_df$total_length_mm)
cat("Mean length of all fish:", round(mean_length, 1), "mm\n")Mean length of all fish: 324.5 mm# Calculate mean by lake
grayling_df %>%
group_by(lake) %>%
summarise(mean_length = mean(total_length_mm, na.rm=TRUE)) %>%
kable(caption = "Mean length by lake", digits = 1)| lake | mean_length |
|---|---|
| I3 | 265.6 |
| I8 | 362.6 |
The spread of a distribution tells us how variable the measurements are.
Variance and Standard Deviation
The variance is
# Calculate standard deviation of fish length
var_length <- var(grayling_df$total_length_mm)
sd_length <- sd(grayling_df$total_length_mm)
cat("Variance of length:", round(var_length, 1), "mm²\n")Variance of length: 4225.9 mm²Standard deviation of length: 65 mm# Calculate by lake
grayling_df %>%
group_by(lake) %>%
summarise(
var_length = var(total_length_mm),
sd_length = sd(total_length_mm)
) %>%
kable(caption = "Standard deviation and variance by lake", digits = 1)| lake | var_length | sd_length |
|---|---|---|
| I3 | 801.1 | 28.3 |
| I8 | 2739.4 | 52.3 |
The area under the curve of a bell shaped curve within + and - 2 Standard deviations on each side includes about 95% of the data
# Read the data
fish_data <- read.csv("data/gray_I3_I8.csv")
# Filter data for I8 lake
i3_data <- fish_data %>%
filter(lake == "I3")
# Calculate statistics
mean_length <- mean(i3_data$total_length_mm)
sd_length <- sd(i3_data$total_length_mm)
# Calculate the bounds for standard deviations
minus_2sd <- mean_length - (2 * sd_length)
plus_2sd <- mean_length + (2 * sd_length)
# Calculate percentage of data within 2 SD
percent_within_2sd <- 100 * mean(
i3_data$total_length_mm >= minus_2sd &
i3_data$total_length_mm <= plus_2sd
)
# Create the plot
ggplot(i3_data, aes(x = total_length_mm)) +
# Add density curve
geom_density(fill = "skyblue", alpha = 0.6) +
# Add vertical line for mean
geom_vline(xintercept = mean_length, color = "navy", linewidth = 1) +
# Add vertical lines for +/- 2 SD
geom_vline(xintercept = minus_2sd, color = "darkred", linewidth = 0.8, linetype = "dashed") +
geom_vline(xintercept = plus_2sd, color = "darkred", linewidth = 0.8, linetype = "dashed") +
# Highlight area within 2 SD
annotate("rect",
xmin = minus_2sd, xmax = plus_2sd,
ymin = 0, ymax = Inf,
fill = "lightgreen", alpha = 0.3) +
# Add annotations
annotate("text",
x = mean_length, y = 0.010,
label = paste0("Mean = ", round(mean_length, 1), " mm"),
color = "navy", fontface = "bold", size = 4) +
annotate("text",
x = mean_length, y = 0.009,
label = paste0("SD = ", round(sd_length, 1), " mm"),
color = "darkred", size = 3.5) +
annotate("text",
x = mean_length, y = 0.008,
label = paste0(round(percent_within_2sd, 1), "% within ±2 SD"),
color = "darkgreen", size = 3.5) +
# Add labels for SD boundaries
annotate("text",
x = minus_2sd, y = 0.002,
label = paste0("-2 SD (", round(minus_2sd, 1), ")"),
color = "darkred", angle = 90, hjust = 0, size = 3) +
annotate("text",
x = plus_2sd, y = 0.002,
label = paste0("+2 SD (", round(plus_2sd, 1), ")"),
color = "darkred", angle = 90, hjust = 0, size = 3) +
# Add title and labels
labs(
title = "Distribution of Fish Lengths in i3 Lake",
subtitle = "Area between dashed lines represents ±2 standard deviations from the mean",
x = "Total Length (mm)",
y = "Density",
caption = paste0("n = ", nrow(i3_data), " fish")
) +
theme_minimal() +
theme(
plot.title = element_text(face = "bold", size = 14),
plot.subtitle = element_text(size = 11),
axis.title = element_text(face = "bold"),
panel.grid.minor = element_blank()
) +
# Set x-axis limits to show the full range plus a bit of padding
xlim(min(i3_data$total_length_mm) - 10, max(i3_data$total_length_mm) + 10)
i3 Lake Fish Length Summary:Number of fish: 66 Mean length: 265.61 mmStandard Deviation: 28.3 mmRange for ±2 SD: 209 to 322.21 mmPercentage within ±2 SD: 90.91 %The coefficient of variation (CV) expresses the standard deviation as a percentage of the mean:
stuff
# Calculate coefficient of variation
cv_length <- sd_length / mean_length * 100
cat("Coefficient of variation:", round(cv_length, 1), "%\n")Coefficient of variation: 10.7 %# Calculate by lake
grayling_df %>%
group_by(lake) %>%
summarise(
cv_length = sd(total_length_mm) / mean(total_length_mm) * 100
) %>%
kable(caption = "Coefficient of variation by lake", digits = 1)| lake | cv_length |
|---|---|
| I3 | 10.7 |
| I8 | 14.4 |
Biological data are often skewed (asymmetrical), which can make the arithmetic mean less representative of central tendency. Data transformations can help address this issue.
Logarithmic Transformation
The logarithmic transformation is one of the most common for right-skewed biological data:
When data are log-normally distributed, the geometric mean often provides a better measure of central tendency than the arithmetic mean.
# Add log-transformed length to our dataset
grayling_df <- grayling_df %>%
mutate(log_mass = log(mass_g))
# Compare original and log-transformed distributions
p1 <- ggplot(grayling_df, aes(x = mass_g)) +
geom_histogram(bins = 15, fill = "steelblue", color = "white") +
labs(
title = "Original Mass Distribution",
x = "Total Mass (g)",
y = "Count"
)
p2 <- ggplot(grayling_df, aes(x = log_mass)) +
geom_histogram(bins = 15, fill = "steelblue", color = "white") +
labs(
title = "Log-Transformed Mass Distribution",
x = "Log(Mass(g)",
y = "Count"
)
# Display side by side
gridExtra::grid.arrange(p1, p2, ncol = 2)
# Compare means on original and transformed scales
mean_log_mass <- mean(grayling_df$log_mass)
back_transformed_mean <- exp(mean_log_mass)
cat("Arithmetic mean of original data:", round(mean_length, 1), "mm\n")Arithmetic mean of original data: 265.6 mmGeometric mean (back-transformed mean of logs): NA mmLecture 3: Visualizing Distributions
Histograms
Histograms show the frequency distribution of our data.
# Create a histogram
ggplot(grayling_df, aes(x = mass_g)) +
geom_histogram(bins = 15, fill = "steelblue", color = "white") +
labs(
title = "Distribution of Fish Mass",
x = "Total Mass (g)",
y = "Count"
)
# Histograms by lake
ggplot(grayling_df, aes(x = mass_g, fill = lake)) +
geom_histogram(bins = 15, position = "dodge", alpha = 0.7) +
labs(
title = "Distribution of Fish Mass by Lake",
x = "Total Mass (g)",
y = "Count"
)
Lecture 3: Visualizing Distributions
Box Plots
Box plots show the median, quartiles, and potential outliers.
# Create a box plot
ggplot(grayling_df, aes(y = total_length_mm)) +
geom_boxplot(fill = "steelblue", alpha = 0.7) +
labs(
title = "Box Plot of Fish Lengths",
y = "Total Length (mm)"
)
# Box plot by lake
ggplot(grayling_df, aes(x = lake, y = total_length_mm, fill = lake)) +
geom_boxplot(alpha = 0.7) +
labs(
title = "Box Plot of Fish Lengths by Lake",
x = "Lake",
y = "Total Length (mm)"
)
Lecture 3: Comparing Mean vs. Mediany
The mean and median measure different aspects of a distribution:
Mean: Center of gravity of the distribution
Median: Middle value of the data
When a distribution is symmetric, the mean and median are similar. When it’s skewed or has outliers, they can differ significantly.
stuff
# Create a density plot with vertical lines for mean and median
ggplot(grayling_df, aes(x = total_length_mm, fill = lake)) +
geom_density(alpha = 0.5) +
geom_vline(data = stats_by_lake,
aes(xintercept = mean, color = "Mean"),
linetype = "dashed", linewidth = 1) +
geom_vline(data = stats_by_lake,
aes(xintercept = median, color = "Median"),
linetype = "solid", linewidth = 1) +
scale_color_manual(values = c("Mean" = "red", "Median" = "blue")) +
facet_wrap(~ lake, ncol = 1) +
labs(
title = "Density of Fish Lengths",
x = "Total Length (mm)",
y = "Density",
color = "Statistic"
)
Demonstrating Sampling Variation
Let’s take several random samples from Lake I3 and see how the sample means vary:
# Filter for Lake I3
i3_data <- grayling_df %>% filter(lake == "I3")
# Function to take a random sample and calculate the mean
sample_mean <- function(data, sample_size) {
sample_data <- sample_n(data, sample_size)
return(mean(sample_data$total_length_mm))
}
# Take 10 different samples of size 15 from Lake I3
set.seed(123) # For reproducibility
sample_size <- 15
sample_means <- replicate(10, sample_mean(i3_data, sample_size))
# Create a data frame with sample numbers and means
samples_df <- data.frame(
sample_number = 1:10,
sample_mean = sample_means
)
# Display the sample means
samples_df sample_number sample_mean
1 1 269.9333
2 2 260.6000
3 3 255.2000
4 4 263.4000
5 5 275.3333
6 6 279.2667
7 7 263.7333
8 8 273.6000
9 9 264.8000
10 10 269.8667[1] 267.5733[1] 7.346063# Plot the different sample means
ggplot(samples_df, aes(x = factor(sample_number), y = sample_mean)) +
geom_point(size = 3, color = "blue") +
geom_hline(yintercept = mean(i3_data$total_length_mm),
linetype = "dashed", color = "red") +
annotate("text", x = 5, y = mean(i3_data$total_length_mm) + 2,
label = "Overall sample mean", color = "red") +
labs(title = "Means of 10 Random Samples from Lake I3",
x = "Sample Number",
y = "Sample Mean (mm)") +
theme_minimal()
Notice how each sample’s mean differs from the overall mean. This demonstrates sampling variation.
Standard Error for Our Grayling Data
Let’s calculate and visualize the standard error for both lakes:
# Calculate mean, SD, and SE for each lake
grayling_stats <- grayling_df %>%
group_by(lake) %>%
summarize(
mean_length = mean(total_length_mm),
sd_length = sd(total_length_mm),
n = n(),
se_length = sd_length / sqrt(n)
)
# Display the statistics
grayling_stats# A tibble: 2 × 5
lake mean_length sd_length n se_length
<chr> <dbl> <dbl> <int> <dbl>
1 I3 266. 28.3 66 3.48
2 I8 363. 52.3 102 5.18# Create a bar plot with error bars representing ±1 SE
ggplot(grayling_stats, aes(x = lake, y = mean_length, fill = lake)) +
geom_bar(stat = "identity", alpha = 0.7) +
geom_errorbar(aes(ymin = mean_length - se_length,
ymax = mean_length + se_length),
width = 0.2) +
labs(title = "Mean Fish Length by Lake with Standard Error",
subtitle = "Error bars represent ±1 standard error",
x = "Lake",
y = "Mean Length (mm)") +
theme_minimal()
Simulating the Sampling Distribution
Let’s simulate taking many samples from Lake I3 to visualize the sampling distribution:
# Filter for Lake I3
i3_data <- grayling_df %>% filter(lake == "I3")
# Number of samples to simulate
num_simulations <- 1000
sample_size <- 20
# Simulate many samples and calculate means
set.seed(456) # For reproducibility
simulated_means <- replicate(num_simulations, sample_mean(i3_data, sample_size))
# Calculate the mean and standard deviation of the simulated means
mean_of_means <- mean(simulated_means)
sd_of_means <- sd(simulated_means)
# Create a data frame with the simulated means
simulated_df <- data.frame(sample_mean = simulated_means)
# Plot the sampling distribution
ggplot(simulated_df, aes(x = sample_mean)) +
geom_histogram(bins = 30, fill = "blue", alpha = 0.7) +
geom_vline(xintercept = mean(i3_data$total_length_mm),
linetype = "dashed", color = "red", size = 1) +
annotate("text", x = mean(i3_data$total_length_mm) + 2, y = 50,
label = "Full sample mean", color = "red") +
labs(title = "Simulated Sampling Distribution of the Mean",
subtitle = paste("Based on", num_simulations, "samples of size", sample_size),
x = "Sample Mean (mm)",
y = "Frequency") +
theme_minimal()Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.
Notice that the simulated sampling distribution:
Is approximately normally distributed
Is centered around the overall sample mean
Has a spread that is related to the standard error
Standard Error and Sample Size
Let’s see how the standard error changes with different sample sizes:
# Define a range of sample sizes to test
sample_sizes <- c(5, 10, 20, 30, 50)
# For each sample size, simulate the sampling distribution and calculate SE
results <- data.frame()
for (size in sample_sizes) {
# Simulate many sample means for this sample size
simulated_means <- replicate(500, sample_mean(i3_data, size))
# Calculate the standard deviation of the sampling distribution (empirical SE)
empirical_se <- sd(simulated_means)
# Calculate the theoretical SE
theoretical_se <- sd(i3_data$total_length_mm) / sqrt(size)
# Add to results
results <- rbind(results, data.frame(
sample_size = size,
empirical_se = empirical_se,
theoretical_se = theoretical_se
))
}
# Display the results
results sample_size empirical_se theoretical_se
1 5 12.349407 12.657835
2 10 8.178270 8.950441
3 20 5.558957 6.328918
4 30 3.792177 5.167540
5 50 2.099744 4.002759# Plot how SE changes with sample size
results_long <- pivot_longer(results,
cols = c(empirical_se, theoretical_se),
names_to = "se_type",
values_to = "standard_error")
ggplot(results_long, aes(x = sample_size, y = standard_error, color = se_type)) +
geom_line() +
geom_point(size = 3) +
scale_x_continuous(breaks = sample_sizes) +
labs(title = "Standard Error vs. Sample Size",
subtitle = "Standard error decreases as sample size increases",
x = "Sample Size",
y = "Standard Error",
color = "SE Type") +
theme_minimal()
Calculating Confidence Intervals for Grayling Data
Let’s calculate and visualize the 95% confidence intervals for the mean fish length in each lake:
# Calculate 95% confidence intervals
grayling_ci <- grayling_df %>%
group_by(lake) %>%
summarize(
mean_length = mean(total_length_mm),
sd_length = sd(total_length_mm),
n = n(),
se_length = sd_length / sqrt(n),
ci_lower = mean_length - 2 * se_length,
ci_upper = mean_length + 2 * se_length
)
# Display the confidence intervals
grayling_ci# A tibble: 2 × 7
lake mean_length sd_length n se_length ci_lower ci_upper
<chr> <dbl> <dbl> <int> <dbl> <dbl> <dbl>
1 I3 266. 28.3 66 3.48 259. 273.
2 I8 363. 52.3 102 5.18 352. 373.# Plot with confidence intervals
ggplot(grayling_ci, aes(x = lake, y = mean_length, fill = lake)) +
geom_bar(stat = "identity", alpha = 0.7) +
geom_errorbar(aes(ymin = ci_lower, ymax = ci_upper),
width = 0.2) +
labs(title = "Mean Fish Length by Lake with 95% Confidence Intervals",
subtitle = "Error bars represent 95% confidence intervals",
x = "Lake",
y = "Mean Length (mm)") +
theme_minimal()
Different Types of Error Bars
Let’s compare different ways of displaying uncertainty in our estimates:
# Calculate statistics for different types of error bars
grayling_error_bars <- grayling_df %>%
group_by(lake) %>%
summarize(
mean_length = mean(total_length_mm),
sd_length = sd(total_length_mm),
n = n(),
se_length = sd_length / sqrt(n),
ci_lower = mean_length - 1.96 * se_length,
ci_upper = mean_length + 1.96 * se_length,
one_sd_lower = mean_length - sd_length,
one_sd_upper = mean_length + sd_length
)
# Create a data frame for plotting different error types
lake_i3 <- grayling_error_bars %>% filter(lake == "I3")
error_types <- data.frame(
error_type = c("Standard Deviation", "Standard Error", "95% Confidence Interval"),
lower = c(lake_i3$one_sd_lower,
lake_i3$mean_length - lake_i3$se_length,
lake_i3$ci_lower),
upper = c(lake_i3$one_sd_upper,
lake_i3$mean_length + lake_i3$se_length,
lake_i3$ci_upper)
)
# Plot the comparison
ggplot() +
geom_point(data = lake_i3, aes(x = "Mean", y = mean_length), size = 4) +
geom_errorbar(data = error_types,
aes(x = error_type, ymin = lower, ymax = upper, color = error_type),
width = 0.2, size = 1) +
labs(title = "Different Types of Error Bars for Lake I3",
subtitle = "Comparing standard deviation, standard error, and 95% confidence interval",
x = "",
y = "Length (mm)",
color = "Error Bar Type") +
theme_minimal() +
theme(legend.position = "none")